System and method for acquiring a still image from a moving image

ABSTRACT

A system and method captures a moving image of a scene that can be more readily de-blurred as compared to images captured through other methods operating on equivalent exposure-time intervals. Rather than stopping and starting the light measurement during the exposure-time interval, photo-generated current is switched between multiple charge storage sites in accordance with a temporal switching pattern that optimizes the conditioning of the solution to the inverse blur transform. By switching the image intensity signal between storage sites all of the light energy available during the exposure-time interval is transduced to electronic charge and captured to form a temporally decomposed representation of the moving image. As compared to related methods that discard approximately half of the image intensity signal available over an equivalent exposure-time interval, such a temporally decomposed image is a far more complete representation of the moving image and more effectively de-blurred using simple linear de-convolution techniques.

RELATED APPLICATION

This application is a continuation-in-part of co-pending U.S. patentapplication Ser. No. 12/559,467, entitled SYSTEM AND METHOD FORACQUIRING A STILL IMAGE FROM A MOVING IMAGE, filed Sep. 14, 2009, andthis application claims the benefit of U.S. Provisional Application Ser.No. 61/793,625, entitled SYSTEM AND METHOD FOR ACQUIRING A STILL IMAGEFROM A MOVING IMAGE, filed Mar. 15, 2013, each of which applications isexpressly incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates to systems and methods for acquiring still imagesfrom moving images.

BACKGROUND OF THE INVENTION

It is often desirable to capture still images of a moving scene with acamera having an electronic imaging system. Conventional electronicimaging systems include a rectangular image sensor, having an array ofelectronic pixels, positioned in the focal-plane of a lens to receive anoptical image of the scene. Electronic pixels include a photosensitiveelement that transduces incident light energy into electronic potential.Most pixels operate by providing for electronic charge to accumulate fora period of time in a storage capacitance. Each storage capacitanceexhibits a voltage potential that is proportional to the total lightenergy arriving at the pixel over the exposure time. In an electronicimage sensor the exposure time must be long enough to generate pixeloutput voltages that are sufficiently distinguishable from the effectsof noise, so as to capture an image that is a useful representation ofthe scene.

A distortion in the captured image, commonly known as motion-blur,occurs when there is a movement in the observed scene that forms anoptical image with respect to the image sensor during exposure time.Unlike a still image where, ideally, the captured pixel values mapuniquely to light emanating from points on observed objects in thescene, in a motion-blurred image, at least some pixel values are afunction of the integral of the light intensities emanating frommultiple points in the scene during the exposure time interval.

A very common approach to the problem of motion-blur is to shorten theexposure time and thereby marginalize the influence of motion on thecaptured image. This is a viable approach to the extent that the opticalimage possesses light energy that is sufficient to satisfy thesignal-to-noise constraint noted generally above. This may require theuse of impractical amounts of illumination at the scene and/or use of alarge lens aperture that adversely affects the image depth of field. Ifthe exposure time can not be shortened sufficiently, an alternativeapproach is to synchronize the motion of the image sensor with themotion of the imaged scene during the exposure time. This approach workswell when a shared external reference frame can be used to facilitatesynchronization, as with, for example, the inertial stabilization ofimage sensors in hand held cameras or the encoder synchronization oftime-delay-integration line-scan cameras.

There are, however, applications where the optical image either does notpossess sufficient light energy to allow for a short exposure timeand/or the motion of the image sensor cannot be practically synchronizedwith the motion of the optical image. In such applications, capturing animage distorted by some amount of motion-blur (also termed herein a“blur image” or “blurred image”) cannot be avoided and a still image canonly be achieved through a computational process that uses a de-blurringalgorithm to extract a still image representation from the capturedimage data.

As described above, motion blurring results when light emanating frompoints on observed objects in the scene that form the optical image movewith respect to the image sensor during exposure time. The patterncreated in the image sensor by one such point source is known as thepoint-spread-function (PSF). The collection of point-spread-functionsacross the image define a blur transform. De-blurring algorithmsestimate the blur transform, solve for the inverse blur transform, andapply the inverse blur transform to the captured image data to arrive ata still image.

Theoretically, point-spread-functions could be completely uncorrelatedacross the image. If this were the case, de-blurring the captured imagewould be virtually impossible. In practice, de-blurring algorithms makenumerous simplifying assumptions. For example, it is common to assumethat the point-spread-function is known and/or spatially invariant overthe image or large segments of the image.

Even given such simplifications, it may be difficult or impossible toarrive at a good solution for the inverse blur transform. This isbecause the motion of the image relative to the image sensor over theexposure period operates as a low-pass box-filter that irretrievablydestroys (or highly attenuates) significant spatial information.Although it is possible to create an approximate inverse to such afilter, the inverse, due to the fact that it is attempting toreconstruct highly attenuated or lost information, becomes verysensitive to input data, which is known as being ill-conditioned. Whenthe solution to the inverse blur transform is ill-conditioned smallchanges in the assumption with regard to how the blurred image wasformed can lead to errors in the resultant still image, derived fromapplying the transform solution, that are hugely out of proportion totheir apparent magnitude in the captured image. Captured imagesinvariably include a noise term that is independent of the blurtransform, and this leads to inaccuracies in the resultant still image.For example, the still image may contain fixed pattern noise or temporalnoise. Fixed pattern noise is often due to dissimilarities of transfergate efficiency or storage capacitance between pixels in the sensor.Temporal noise often occurs due to shot noise in the photodetector oramplifier noise. De-blurring algorithms typically manage this problem byeither attempting to filter the independent noise component in thecaptured image and/or forcing a modification on thepoint-spread-function that improves the conditioning of the inverse blurtransform.

Most de-blurring algorithms do not assume any ability to influence theoriginal blur transform. Such algorithms simply operate, to variousextents, with a digital input image and some very general a prioriexpectations regarding image noise. Such algorithms are, in effect,restricted to attempting to separate the influence of noise from anill-conditioned solution to the inverse blur transform, which is, initself, an extremely difficult problem.

It should be noted that the art provides a class of algorithms that arecapable of estimating the image motion as input, and this estimate canbe employed to aid in deriving a still image by various de-blurringalgorithms. For example, Ben Ezra et. al. employ a secondary motiondetector to estimate image motion, and employ this estimate to aid inderiving a still image, as described, by way of background inMotion-Based Motion Deblurring, Moshe Ben-Ezra and Shree K. Nayar, IEEETRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO.6, June 2004.

Other, more-recently developed, de-blurring algorithms incorporatecontrol of the image exposure process. One such algorithm includes theuse of multiple exposures, where by the total image exposure time islong enough to achieve good signal-to-noise in the captured image andthe induced modification of the point-spread-function leads to aninverse blur transform that is optimal in terms of conditioning. Amethod for finding such a multiple exposure pattern includes evaluatingvarious exposure patterns until the pattern that results in asufficiently small condition number in the inverse blur transform isfound. The multiple exposure process itself includes repeatedly stoppingand starting the integration of light measurement over an exposure-timeinterval. A description of one such method is given in U.S. PublishedPatent Application No. 2007/0258706 A1, entitled METHOD FOR DEBLURRINGIMAGES USING OPTIMIZED TEMPORAL CODING PATTERNS, by Raskar, et al., andrelated applications thereto, the teachings of which are expresslyincorporated herein by reference as useful background information.

One disadvantage of the method described above is that for the modifiedpoint-spread-function to have a substantial influence on theconditioning of the inverse blur transform, it should incorporatenumerous exposure activation periods of varying duration separated by asimilar set of exposure deactivation periods. As the size of thepoint-spread-function grows, so do the chances that one or moresimplifying assumption may be violated. For example, it is common toexpect the velocity of the scene, or of objects in the scene, to beconstant over the exposure period—as the duration of the period isrelatively short and change may occur in a longer time than the durationof the exposure period. As the duration of the exposure time increasesthe robustness of a constant velocity assumption decreases. With a largepoint-spread-function the inverse blur transform is more influenced byboundary conditions, meaning that it is more dependent on the state ofpixels that are outside the field-of-view of the captured image, ascompared to a smaller point-spread-function. In addition, the accuratede-blurring of objects in the scene having a different velocity thantheir immediate surroundings may be restricted by the relative size ofthe point-spread-function, since the spatial invariance simplificationwill be violated on a boundary that is related in size to thepoint-spread-function. That is, the point-spread-function is notconsistent for elements of the image that are not moving versus elementsthat are moving, and thus, a point-spread-function that applies to asmaller-neighborhood is desirable to reduce the effects of boundaryconditions.

Another disadvantage to certain prior art deblurring algorithms, andparticularly those employing intermittent starting and stopping ofintegration, is that the amount of light in a given exposure period isreduced. The lens aperture must generally be increased to satisfy thesignal-to-noise constraints of the sensor, thereby providing sufficientlight to the sensor to form an acceptable image within the integrationperiod. Increasing the aperture can adversely affect the depth of fieldof the imaging system.

A system capable of capturing a readily de-blurred image of a movingscene while operating on a significantly shorter time interval, ascompared to known methods, would be desirable. The ability to capturesuch an image employing a reduced aperture for enhanced depth of fieldis also desirable.

SUMMARY OF THE INVENTION

This invention overcomes the disadvantages of prior art known methodsfor capturing a moving image. Such known methods for capturing a movingimage that can be more readily de-blurred as compared to images capturedusing typical techniques include capturing a multiple exposure image byrepeatedly starting and stopping the integration of light intensitymeasurement over an exposure-time interval. This can be accomplished byvarious methods, including modulating an electronic shutter in atemporal pattern selected to improve the conditioning of the solution tothe inverse blur transform of the captured image. Such multiple exposurepatterns are characterized by an irregular sequence of integration andnon-integration periods over the exposure-time interval, the totalamount of integration time and non-integration time being approximatelyequal.

According to an illustrative embodiment, the system and method capturesa moving image of a scene that can be more readily de-blurred ascompared to images captured through the above-referenced and other knownmethods operating on an equivalent exposure-time interval. Rather thanstopping and starting the integration of light measurement during theexposure-time interval, photo-generated current is switched betweenmultiple charge storage sites in accordance with a temporal switchingpattern that optimizes the conditioning of the solution to the inverseblur transform. By switching the image intensity signal betweenintegrating storage sites substantially all of the light energyavailable during the exposure-time interval is transduced to electroniccharge and captured to form a temporally decomposed representation ofthe moving image. As compared to related methods that discardapproximately half of the image intensity signal available over anequivalent exposure-time interval, such a temporally decomposed image isa far more complete representation of the moving image and moreeffectively de-blurred using known and typically straightforward linearde-convolution techniques.

In an illustrative embodiment a system and method for acquiring a stillimage from an input moving image, provides an imaging assembly includinga plurality of pixels. Each of the pixels comprises a photosensitiveelement and a plurality of integrating storage sites, and each of thepixels is constructed and arranged to direct a measurement of lightimpinging on the photosensitive element to any one of the plurality ofintegrating storage sites. An image capture process forms a temporallydecomposed representation of the moving image in an exposure timeinterval by repeatedly switching the measurement of light impinging oneach of the photosensitive elements among the plurality of integratingstorage sites according to a temporal switching pattern. An imageextraction process operates on the temporally decomposed representationof the moving image to extract the still image. The photosensitiveelement is illustratively a photodiode connected to a first sense nodethrough a first transfer gate and to a second sense node through asecond transfer gate, and each of the first transfer gate and the secondtransfer gate are responsive to the input image capture process. Thetemporal switching pattern can be selected to (a) minimize attenuationof spatial frequencies in the temporally decomposed representation ofthe moving image; (b) minimize attenuation of spatial frequencies in thestill image; (c) provide a still image that contains a filtered set ofspatial frequencies; and/or (d) provide a still image having reducedblur.

In an illustrative embodiment, the image extraction process isconstructed and arranged to provide a point-spread-function having aplurality of coefficients based upon a linear combination of valuesstored in the each of the plurality of integrating storage sites. A blurmatrix is provided based upon the point-spread-function. This blurmatrix is illustratively based upon a best version of the temporalswitching pattern in terms of the residual error and the conditionnumber. The system and method cycles through a plurality of candidatetemporal switching patterns in order to identify the one with theminimal residual error and condition number values. This is chosen asthe best temporal switching pattern.

Additionally, the moving image can be acquired from a moving object orscene operatively connected with an encoder that provides a signalrelated to movement of the object or scene, and the temporal switchingpattern can be computed based upon the signal. An object-heightdetection assembly can also be operatively interconnected between theencoder and an image switch signal generator to modify the temporalswitching pattern based upon the actual height of the object and itsassociated proximity to in image sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a diagram of a system and method for capturing a still imagefrom a moving image including a camera and associated functionalcomponents acquiring an image of an exemplary object moving through itsfield of view on an exemplary conveyor;

FIG. 2 is a block diagram representing one electronic pixel of imagesensor of the illustrative camera of FIG. 1;

FIG. 3 is a flow diagram of a procedure for finding the best switchingpattern for a point-spread-function of known length for use in theillustrative system and method;

FIG. 4 is a diagram showing an illustrative solution for an inverse blurtransform used in conjunction with the illustrative system and method;

FIG. 5 is a flow diagram showing the overall procedure for acquiring astill image from a moving image in accordance with an illustrativeembodiment;

FIG. 6 is a diagram showing an illustrative solution for an inverse blurtransform for use in computing a still image from a moving image, inaccordance with an illustrative embodiment;

FIG. 6A is graphical diagram showing the plot of the inverse blur, inaccordance with an illustrative embodiment;

FIG. 6B is a graphical diagram showing a plot of the blur multiplied bythe inverse, in accordance with an illustrative embodiment; and

FIG. 7 is a flow diagram showing a procedure for finding the besttemporal switching pattern for the point-spread-function, in accordancewith the illustrative embodiment.

DETAILED DESCRIPTION

FIG. 1 shows schematic camera 106, representative of one illustrativeembodiment of the present invention. Camera 106 is focused on a point103 of object 102, which rests on moving conveyor 101. Camera 106includes lens 107, image switch signal generator 108, image sensor 110,image processor 115, and image storage buffer 116. Image sensor 110includes a light intensity transducer 111, an image switch 112, anintegrating image-storage buffer 113, and another integratingimage-storage buffer 114.

In operation, and with further reference to the flow diagram showing theoverall procedure 500, prior to the exposure-time interval, integratingimage-storage buffers 113 and 114 (or equivalent integrating storagesites) are reset (step 510). The process 500 then initiates an imagecapture process as shown by the dashed box in FIG. 5. During theexposure-time interval, some of the light reflected from moving object102 is collected by lens 107 and focused to form a moving image on thefocal plane of image sensor 110 (step 512). As defined herein, a “movingimage” is a three-dimensional representation of the object within theimage sensor in which the third dimension is the exposure time and theimage exhibits spatial changes with respect to time. In the depictedscenario of FIG. 1, the motion of the object 102 over the exposure timeof the sensor 110 causes the creation of a moving image. However, avariety of alternate scenarios can cause creation of a moving image.

Light from the moving image is converted by light intensity transducer111 into an array of electrical signals proportional to incident lightintensities. The transducer can be organized as an array of electronic“pixels”, arranged in either a one-dimensional or two-dimensionalconfiguration. Instantaneous or near-instantaneous measurements of lightimpinging upon the photosensitive elements of each of the pixels in thetransducer 111 (the term “instantaneous” herein taking into accountdelays based upon inherent electronic and quantum effects) provide aphotocurrent that is directed as a signal among any of the integratingstorage sites/buffers 113 and 114 depending upon the state of thetransfer gates. Thus, the light intensity signals output by lightintensity transducer 111 are switched repeatedly in parallel, by imageswitch 112, between integrating image-storage site/buffer 113 and 114,in accordance with the temporal switching pattern input from imageswitch signal generator 108 over the exposure time interval (step 514).In this manner, the moving image is temporally decomposed between atleast two integrating storage sites or buffers. The temporal switchingpattern (or simply, “switching pattern”) is optimized, being typicallycomputed as described in accordance with FIG. 3 below. An optional shaftencoder 105, or other motion-sensing device is operatively connectedwith the conveyor belt (and/or drive mechanism), and a trigger signal109 may be provided to synchronize image switch signal generator 108with moving object 102.

When image exposure is complete (decision step 516), image processor 115operates an image extraction process (dashed box in FIG. 5) thatconvolves the temporally decomposed representation of the moving imagecaptured in integrating image-storage buffer 113 and 114 withcoefficients of the inverse blur transform to extract and form a stillimage in image storage buffer 116 (step 518). This “still image”desirably defines a two-dimensional image where the time dimension ofthe exposure has been collapsed or minimized so as to render the resultsufficiently de-blurred.

FIG. 2 is a block diagram representing one electronic pixel of imagesensor 110 according to an illustrative embodiment. Pixel 200 includes,a photosensitive element in the form of a pinned photodiode 201, twotransfer gates 202 and 212, two floating diffusion nodes 203 and 213,two reset transistors 204 and 214, two source-follower transistors 205and 215, two read-select transistors 206 and 216, and one anti-bloomingtransistor 207. A double-sampling amplifier 209 samples the pixel outputvoltage at transistor 208, once when diffusion 203, 213 is in the resetstate and once after diffusion node 203, 213 is charged by pinnedphotodiode 201. The difference of the two voltages is computed (i.e.double-sampled) to arrive at a corrected pixel output voltage. Thecorrected output voltage is digitized by analog-to-digital converter 210so as to provide a digital intensity value for the pixel.

Pinned photodiode 201 corresponds to one discrete photosensitive element(transducer) of the overall row-and-column pixel array of the imageintensity transducer 111. Pinned photodiode 201 converts incident lightenergy into photocurrent proportional to the light intensity. Togethertransfer gates 202 and 212 correspond to one switch of image switch 112.Floating diffusion nodes 203 and 213 each correspond to one storage siteof integrating image-storage buffer 113 and 114, respectively.

During the image exposure time interval the image switch signalgenerator 108 drives all transfer-gates 202 and 212 of image sensor 110,repeatedly turning each of the two transistors on and off according to apredetermined pattern. In typical operation, the exposure-time switchingpattern is complementary (i.e. when one transfer gate is on the other isoff). Note that a time interval may also exist when both gates are on oroff during the switching process, which may be ignored by the processorhardware and software.

The switching pattern can be governed by a variety of factors. Switchingis desirably configured to allow a given object feature to be acquiredfully within the switching interval—i.e. without loss of featureinformation. For example, where a feature is one pixel wide in thecaptured image, it may be desirable to provide a switching interval thatchanges during object or scene motion of a half pixel distance or less.Where the feature is several pixels wide, then the switching intervalcan change after more than one pixel distance of object motion. Anexemplary feature that can be imaged in accordance with an illustrativeembodiment is a barcode pattern in which the code elements may vary by apixel distance or less, thus making a shorter switching intervaldesirable.

Briefly described, after image exposure, the charge accumulated infloating diffusion node 203 is primarily the result of photocurrentgenerated by photodiode 201 when gate 202 was open, while the charge infloating diffusion node 213 was accumulated when transfer gate 212 wasopen. In the illustrative embodiment, voltage potentials on floatingdiffusion nodes 203 and 213 are digitized by image processor 115 andconvolved with coefficients of the inverse blur transform to form adigital representation of the still image that is stored in image buffer116.

FIG. 4 is a diagram showing an exemplary solution for an inverse blurtransform for use in computing a still image from a moving image inaccordance with the illustrative embodiment. For a given amount of lightintensity, the image formed on image sensor 110 will move a pixeldistance that is proportional to the required exposure time. By way ofan example, it is assumed that the image moves a distance of 10 pixelsalong the row axis of the image sensor during the exposure timeinterval. In this case the point-spread-function will have a length of10 pixels. Assuming that the image velocity is constant over theexposure time, then incident light from any and every single point inthe scene will be divided evenly between 10 contiguous horizontalpixels. Let the symbol “A” represent integrating storage buffer 113, letthe symbol “B” denote integrating storage buffer 114, and let theswitching control pattern be described by an array consisting of somenumber of symbols “A” and an equal number of symbols “B”. For thisexample let the switching control be defined by the pattern 401“AAABBBABBA”. The point-spread-function is created by linear combinationof the contents of integrating storage buffer 113 with the contents ofintegrating storage buffer 114. Although in theory any coefficients arepossible, it has been determined through practice that for a balancedswitching pattern such as 401 “AAABBBABBA”, where the captured image isdistributed equally between two storage buffers, the coefficients 2 and−1 yield good results. Therefore, in this example switching pattern 401“AAABBBABBA” yield point-spread-function 402 {+2, +2, +2, −1, −1, −1,+2, −1, −1, +2}. Assuming zero boundary condition, thepoint-spread-function 402 is padded with zeros and expanded to form thedepicted Toeplitz blur matrix 403. The blur transform is computed bysolving for the coefficients 404 that provide the minimum RMS residualerror fit (step 305 below) to an impulse function, which in this exampleis scaled by 100-times (100×) to facilitate a fixed point implementationof the de-blurring process. The current result 405 of the multiplicationof the blur matrix 403 with the blur transform coefficients 404 is shownas column 405, which is the impulse response scaled by 100-times.

More generally, an explanation of fundamental algebraic spatial imagereconstruction techniques can be found by way of useful background inDigital Image Processing, by William K. Pratt, John Wiley & Sons, 1978.This text provides a complete survey of classical image reconstructiontechniques that should be familiar to those of ordinary skill.

In summary, an applicable technique in accordance with an embodiment isdescribed as:

In conventional notation the reconstruction problem is stated:B=AX+E

By way of explanation of the above equation, the acquired “blurred”image B is decomposed as product of the “true” image X multiplied by theblur matrix A plus an acquisition error matrix E. This implies that ifone knows the blur matrix A, it is possible to solve for the “true”image X, as follows.X=A′B−A′E

One goal, given some limited a priori knowledge regarding thecomposition of the error matrix E, is to render A′ (the inverted blurmatrix) less likely, when multiplied by E, to dominate A′B in thesolution for the true image X. This is achieved in the manner describedfurther below.

Unlike the various traditional image reconstruction techniques describedin Pratt above, the construction of the illustrative blur matrix is notlimited to positive values, and, instead, includes both positive andnegative coefficient values. This is made practical by way of the highspeed temporal decomposition of the point-spread-function enabled by theillustrative switching technique herein.

It has also been observed that a beneficial influence on the error termA′E can be obtained as a result of forming the blur matrix A with bothpositive and negative values. The benefit can be observed in theconditioning of the solution for A′ which is indicative of lowersensitivity to high frequency noise associated with typical forms ofimage acquisition error.

FIG. 3 is a flow chart depicting a method 300 for finding the besttemporal switching pattern for the point-spread-function (for example,above-described PSF 402) of known length. As in the example above,assume that the image is known to move a distance of 10 pixels duringthe exposure time and that the switching pattern should be balanced, soas to evenly distribute the integration of incident light measurementbetween two integrating storage sites/buffers. The best switchingpattern will be the pattern that provides for the best solution to theinverse blur transform, in terms of the condition number and residualerror. One method for finding such a switching pattern includesinitiating an outer loop 301 that cycles through all possible switchingpatterns consistent with the length and balance constraints. Briefly,within outer loop 301: step 302 uses the candidate switching pattern togenerate a blur matrix; step 303 solves for the blur transformcoefficients; step 304 computes the condition number and residual errorof the solution; decision steps 305 and 306 compare the current residualerror and condition number to the previously existing minimum values todetermine if the current switching pattern represents a better solution.If a better solution is found, then the switching pattern along with theresidual error and condition number metrics are saved in step 307, andouter loop 301 continues via decision step 308 until all switchingpatterns have been evaluated. When all patterns have been evaluated, themethod returns the best pattern from step 307 via decision step 308.

By way of further background, the condition number is known in the artas the sensitivity of the solution to noise. A solution with a lowcondition number is said to be well-conditioned, while a solution with ahigh condition number is said to be ill-conditioned. This number can becomputed by known techniques.

It has been observed that the above-described technique for computing ade-convolution kernel provides for a more-limited-neighborhood (i.e.less than the entire image data available) de-convolution kernel thatresults in lower residual error for less computational effort and,generally, three-times or better conditioning, as compared to a similarsolution based on a binary exposure coding pattern of similar size. Theswitching pattern can be computed for each object or scene passingthrough the sensor's field of view as described above, or can becomputed more-intermittently, where objects are likely to exhibit thesame velocity and feature characteristics, with the previously computedswitching pattern being stored and reused with each image capture andimage extraction process.

The size of the point-spread-function (based upon pixel distance orother measurement units) is computed using a variety of techniques.Typically, the size is computed based upon the velocity of the object orscene. Based upon the velocity, the number of pixels passing a point ina given exposure time can be computed. Typically, the velocity iscomputed by (a) reading the encoder 105, (b) by a predeterminedestimation of velocity and/or or (c) by employing an image-baseddetermination of object motion. A number of algorithms exist in the artfor estimating motion of an object using captured image data. Forexample, a conventional implementation of a Kalman filter can beemployed to predict the prevailing object speed, based upon the changein the position of predetermined image data versus time in the periodpreceding the application of the Kalman filter computation. In yetanother implementation, the motion of the belt or object can be readbased upon dynamic feature detection. That is, the variations in thesurface geometry of the belt or object adjacent to the features ofinterest (for example, defects in the object or belt surface, printfeatures, grain boundaries, etc.) can be employed to define a prevailingspeed of the object through the camera field of view. A description ofdynamic feature detection is provided in commonly assigned U.S. patentapplication Ser. No. 12/100,100, entitled METHOD AND SYSTEM FOR DYNAMICFEATURE DETECTION, filed Aug. 9, 2008, by William M. Silver, theteachings of which, along with the and other incorporated US Patentapplications therein, are expressly incorporated herein by reference asuseful background information. The point spread function can beotherwise defined as a constant value based upon previously knowninformation about object features and motion.

As described generally above, the computation of an optimized temporalswitching pattern can take into account one or more goals. In general, adisadvantage of prior image deblurring techniques is that they result inattenuation of spatial frequencies that may cause the loss of neededinformation in the resulting still image. Thus, the temporal switchingpattern should be selected for a variety of reasons. One is to minimizeattenuation of spatial frequencies in the temporally decomposedrepresentation of the moving image so information will be retained inthe still image. The image extraction process can also minimize theattenuation of spatial frequencies in the still image. Likewise, theimage capture process and the image extraction process can provide astill image that contains a filtered set of spatial frequencies. Asanother goal, the image capture process and the image extraction processcan provide a still image having reduced blur.

Using an encoder or other physical measurement of instantaneous (ornear-instantaneous) velocity, it is also contemplated that the switchingpattern can be computed based upon the actual velocity curve forincreased accuracy. For example, instantaneous velocity of the object orscene is known, movement of pixel distance per-unit-time can be computedand the switching interval can thereby be determined to ensureappropriate correspondence of the switching between storage sites withthe motion of object features in the moving image in the presence of avarying velocity profile (such as acceleration or deceleration ofconveyor belt 101).

It is contemplated that the height of the object 102 can vary, andthereby affect the relative pixel distance if features. This, in turn,affects the computation of the temporal switching pattern. For example,the features of a closer object to the sensor appear larger and occupy agreater pixel distance than a further-away object. Thus, the encoder 105can operate in conjunction with an optional object-height-detectionassembly 130 (FIG. 1) according to an illustrative embodiment. A varietyof known mechanisms can be employed to determine object height includingdepicted light curtain in which the reflected or transmitted beamsdetermine the approximate height of the object's camera-facing surface.Alternate height detection mechanisms can include sonar devices andoptical range finders. A scaling processor 132 receives inputs from theheight-detection assembly and the encoder, and modifies the input signalto the image switch signal generator 108 to account for the detectedheight of the object so that the switching pattern is modified in viewof the actual pixel distance versus object motion.

It should be noted that the use of an encoder or other movementmeasurement device operatively connected to the object or scene can beapplied to more-conventional shutter coding or image modulationarrangements according to an embodiment. Illustratively, a system andmethod for extracting a still image from a moving image of a movingobject or scene acquired by a sensor entails the switching the lightimpinging on a photosensitive element of each of a plurality of pixelsof the sensor reflected from the moving object or scene so as tomodulate photocurrent integrated by a storage site according to anoptimal temporal switching pattern during an exposure time to generatean encoded image. The temporal switching pattern is computed by readingthe movement signal provided by the encoder or other operativelyconnected movement sensing device so as to derive a relativelycontemporaneous measurement of object motion. In this embodiment, theswitched intervals can alternate between storing the intensity value inan integrating buffer and discarding or ignoring the intensity value. Aprocessor then decodes the encoded image according to an inverse of thetemporal switching pattern to provide a still image.

FIG. 6 is a diagram showing an illustrative solution for an inverse blurtransform for use in computing a still image from a moving image, inaccordance with an illustrative embodiment. For a given amount of lightintensity, the image formed on the image sensor (for example 110 inFIG. 1) will move a pixel distance that is proportional to the requiredexposure time. By way of an example, it is assumed that the image movesa distance of 10 pixels along the row axis of the image sensor duringthe exposure time interval. This value is exemplary in alternateimplementations. In this illustrative case, the opticalpoint-spread-function will have a length of 10 pixels. Assuming that theimage velocity is constant over the exposure time, then incident lightfrom any and every single point in the scene will be divided between 10contiguous horizontal pixels. With reference to FIG. 6, let the symbol“A” represent a first integrating storage buffer (for example 113 inFIG. 1) and let symbol “B” denote a second integrating storage buffer(for example 114 in FIG. 1). Further, let the switching control patternbe described by an array consisting of some number of symbols “A” and anumber of symbols “B”. For this example, the switching control isdefined by the pattern 601 “BBBABBAAAA”. The switching patterntemporally decomposes the moving image into a 3-dimensional tensorhaving the height and width of the image pixel array and depth of twoelements. The first element of the depth mode is an image matrix formedby temporal sampling according to pattern 602 a [0001001111] and thesecond element of the depth mode is a second image matrix formed bysampling according to pattern 602 b [1110110000]. Assuming zero boundarycondition, the point-spread-function 602 is padded with zeros andexpanded to form a blur tensor, which is unfolded about mode-1 to formthe blur matrix, depicted in 603 as black representing zeros and whiterepresenting ones. Note that the entire blur 603 is shown. However, inapplication, only 1 row is taken, as shown by the dotted-line box 603 aindicating a single row that is analyzed at a time. In runtimeoperation, the system performs the exemplary inverse blur transformrow-by-row until the entire image has been processed.

It should be clear to those of skill that the problem illustrated inFIG. 6 is “underdetermined”, meaning that there are more unknowns thanthere are equations to solve for the unknowns. In general, this meansthat there can be infinitely many exact solutions to the inverse blur.However, although many exact solutions exist, some solutions are betterthan others. In a practical implementation of the present invention,conditioning of the solution to the inverse blur is of paramountimportance because all image sensors induce some amount of additivenoise, which is substantially random in nature and therefore cannot beperfectly accounted for in the image reconstruction process. Suchadditive image sensor noise can be greatly amplified in the imagereconstruction process by a poorly conditioned inverse blur function,rendering the de-blurred image extremely noisy in terms of theprevailing signal-to-noise ratio.

To avoid the foregoing problem of additive noise inherent in all imagesensors, the illustrative system and method compute the inverse blur bysearching for the coefficients 604 in accordance with the statement 605.It is desirable to minimize the L1-norm (min∥Inverse∥₁) of the inverseblur 604, subject to the constraint that the L2-norm(∥Blur*Inverse−Impulse∥₂≦ε) of the residual error remain below someminimum value epsilon. The minimum value epsilon (ε) is chosen inaccordance with the expected level of image sensor noise. The L1-normand L2-norm are computed using techniques known to those skilled in theart to solve for the unknowns. The L1-norm and L2-norm provide the totallength of all vectors in a vector space or matrix. The forgoing solutioncan be performed by interior points methods or other methods known tothose skilled in the art. The statement 605 is one column of outputthat, when combined with the analysis for each row of the image 603,provide the overall desired result.

Reference is now made to FIGS. 6A and 6B showing graphical diagrams ofthe plot of the inverse blur and the blur multiplied by the inverse,respectively, in accordance with an illustrative embodiment. FIG. 6Ashows a plot of the inverse blur 604 a for the 10 element switchingpattern “BBBABBAAAA” which has been found to be optimum, according tothe method 700 shown and described in greater detail herein. It can beobserved from the plot 605 a of (Blur*Inverse) that some residual errorexists, consistent with the epsilon tolerance constraint. It can befurther observed that the inverse blur 604 a is relatively sparse andthat its coefficients are small, consistent with the minimization of theL1-norm, leading to minimum sensitivity of the inverse blur to imagesensor noise.

FIG. 7 is a flow diagram showing a procedure for finding the besttemporal switching pattern for the point-spread-function, in accordancewith the illustrative embodiments. The pint-spread-function can be theillustrative point-spread-function 602 shown and described hereinaboveand having a known length. For descriptive purposes, as in theillustrative embodiment hereinabove, assume that the image is known tomove a distance of 10 pixels during the exposure time and that theswitching pattern should be balanced, so as to evenly distribute theintegration of incident light measurement between two integratingstorage sites/buffers. The best switching pattern is the pattern thatprovides the best solution to the inverse blur transform and that iswell-conditioned (i.e. in terms of condition number and/or residualerror). One method for finding a well-conditioned switching pattern isshown in FIG. 7. The illustrative procedure commences at step 701 tocycle through all possible switching patterns consistent with the lengthand balance constraints. Briefly, within the outer loop 701: step 702uses the candidate switching pattern to generate a blur matrix; step 703solves for the inverse coefficients; decision steps 704 compares thecurrent L1-norm of the inverse blur to the previously existing minimumvalues to determine if the current switching pattern represents a bettersolution. If a better solution is found, then the switching patternalong with the new L1-norm minimum value are saved in step 705, andouter loop 501 continues via decision step 706 until all switchingpatterns have been evaluated. When all patterns have been evaluated, themethod returns the best pattern from step 705 via decision step 706.

It is expressly contemplated that the system and method described abovecan be implemented so that stored moving image data in the plurality ofimage storage sites is subject to the image extraction process, orportions thereof, at a time subsequent to image capture, potentiallyusing a remote image processor. The appropriate parameters forcomputation of the point-spread-function are stored to enable subsequentcomputation of the blur transform and derivation of the still imagetherefrom.

Likewise, it is expressly contemplated that the architecture of thetransducer array and associated pixels can be varied, relative to thearchitecture depicted in FIG. 2, which is based generally on a5-transistor CMOS pixel. In an alternate embodiment, the pixel can beconstructed according to a different architecture. For example, theinterconnected array can be addressed according to a differentconnectivity. The diffusion nodes can also be gated by a number ofcircuit arrangements. The array can be organized with a plurality ofpixels in a variety of one-dimensional or two-dimensional arrangements.Where a one-dimensional arrangement is employed, the pixels shouldtypically be oriented along the prevailing direction of motion so as toensure capture of the motion image. In alternate embodiments, the pinnedphotodiode can be otherwise arranged to provide two or more discretesignals to discrete integrating storage sites. Likewise, while theswitching pattern is described as including two states corresponding toa signal directed to either of the two integrating storage sites, it iscontemplated that three or more switching pattern states can exist,directed to three or more integrating storage sites.

It is understood that there may exist time periods during the switchingbetween integrating storage buffers in which the signal is received atboth storage buffers, or neither storage site receives a signal due todue to latency in the electronic components. It is also understood thatone may deliberately introduce a period in which the signal does notreach any integrating storage site, or is not read. However, theprovision of two or more integrating storage sites or buffers,notwithstanding such inherent or deliberate introduction of anadditional period in which no stored signal exists, still provides anovel arrangement according to the teachings of this invention.

Also, the image transducer, integrating storage sites/buffers and otherelectronic hardware components described herein can be implementedvariously as separate components joined by leads or cabling, or as partof a single chip or chipset as desired. Such a single chip or chipset(not shown) can include other vision processing components for storingand manipulating object or scene feature information.

In summary, it should be clear that the above-described system andmethod provides various advantages over prior deblurring solutions. Byswitching between two or more integrating storage sites, the amount oflight captured from the object or scene is substantially increased whencompared with single-storage site systems that employ shutter coding inthe acquisition of the moving image. This allows for reduced exposuretime, a smaller aperture for greater depth of field, and/or an increasedlikelihood that a constant velocity assumption for the object or sceneis valid. Moreover, the resultant reduction in size of thepoint-spread-function array allows for less computational overhead incomputing the blur transform and deriving the extracted still image.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention. Eachof the various embodiments described above may be combined with otherdescribed embodiments in order to provide multiple features.Furthermore, while the foregoing describes a number of separateembodiments of the apparatus and method of the present invention, whathas been described herein is merely illustrative of the application ofthe principles of the present invention. For example, a variety ofadditional filtering processes and statistical smoothing functions canbe applied to the image data to further enhance the fidelity/sharpnessof the estimated original image. In one example, the image data issubjected to multiple convolution steps in the image extraction process.Moreover, while the illustrative embodiment depicts blur defined in themoving image mainly as a result of object motion, it is expresslycontemplated that the principles herein are applicable to camera motionrelative to the object and/or a combination of object and camera motion.In general, the teachings herein can be performed using hardware,software that comprises computer-readable program instructions, or acombination of hardware and software. Accordingly, this description ismeant to be taken only by way of example, and not to otherwise limit thescope of this invention.

What is claimed is:
 1. A system for acquiring a still image from aninput moving image, comprising: a) an imaging assembly including aplurality of pixels, each of the pixels comprising a photosensitiveelement and a plurality of integrating storage sites, each of the pixelsbeing constructed and arranged to direct a measurement of lightimpinging on the photosensitive element to any one of the plurality ofintegrating storage sites; b) an image capture process that forms atemporally decomposed representation of the moving image in an exposuretime interval by directing, according to a predetermined temporalswitching pattern, the measurement of light impinging on each of thephotosensitive elements among the plurality of integrating storagesites, such that the temporally decomposed image is formed with atemporally decomposed point-spread-function having an inversetransformation that is optimally well-conditioned; and c) an imageextraction process that applies the inverse transformation of thetemporally decomposed point-spread-function to the temporally decomposedimage to form a motion-blur free digital image.
 2. The system as setforth in claim 1 wherein the photosensitive element is a photodiodeconnected to a first sense node through a first transfer gate and to asecond sense node through a second transfer gate, each of the firsttransfer gate and the second transfer gate being responsive to the inputimage capture process.
 3. The system as set forth in claim 1 wherein theplurality of integrating storage sites include at least a thirdintegrating storage site, each of the pixels being constructed anarranged to direct the measurement of light impinging on thephotosensitive element to the at least third storage site according tothe temporal switching pattern.
 4. The system as set forth in claim 1wherein the temporal switching pattern is selected to at least one of(a) minimize attenuation of spatial frequencies in the temporallydecomposed representation of the moving image; (b) minimize, by theimage extraction process, attenuation of spatial frequencies in thestill image; (c) provide, by the image capture process and the imageextraction process, a still image that contains a filtered set ofspatial frequencies; and (d) provide, by the image capture process andthe image extraction process a still image having reduced blur.
 5. Thesystem as set forth in claim 1 wherein the image extraction process isconstructed and arranged to provide a point-spread-function having aplurality of coefficients based upon a linear combination of valuesstored in the each of the plurality of integrating storage sites.
 6. Thesystem as set forth in claim 5 wherein the image extraction process isconstructed and arranged to provide a blur matrix based upon thepoint-spread-function, the blur matrix being based upon a best versionof the temporal switching pattern in terms of a residual error and acondition number.
 7. The system as set forth in claim 6 wherein theimage extraction process is constructed and arranged to determine thebest version of the temporal switching pattern by (a) using each of aplurality of candidate temporal switching patterns to respectivelygenerate the blur matrix; (b) solving for a set of blur transformcoefficients with respect to each blur matrix respectively; (c)computing respectively the residual error and the condition number withrespect to each set of blur coefficients; (d) comparing a current valueof the residual error and a current value of the condition number to apreviously existing minimum value of the residual error and a previouslyexisting minimal value of the condition number to determine if a currentone of the candidate temporal switching patterns represents a bettersolution for at least one of the residual error and the conditionnumber, and if the current one of the candidate temporal switchingpatterns represents a better solution, saving the current one of thecandidate switching patterns in association with the current value ofthe residual error and the current value of the condition number; (e)and, when all of the candidate temporal switching patterns have beenused, returning the best version of the temporal switching pattern. 8.The system as set forth in claim 5 wherein the coefficients include bothpositive and negative values.
 9. The system as set forth in claim 1wherein the moving image is acquired from a moving object operativelyconnected with an encoder that provides a signal related to movement ofthe object or scene, the temporal switching pattern being based upon thesignal related to the movement.
 10. The system as set forth in claim 9,further comprising an object-height-detection assembly constructed andarranged to measure a height of a surface facing the imaging assembly,and based upon the height, varying the temporal switching pattern inconjunction with the signal related to movement of the object providedfrom the encoder.
 11. The system as set forth in claim 10 wherein theheight-detection-assembly comprises a light curtain.
 12. The system asset forth in claim 10 wherein the height-detection-assembly isconstructed and arranged to measure the height based upon at least oneof an optical and a sonar measurement.
 13. The system as set forth inclaim 1 wherein the moving image defines a blurred image and whereinblur therein is caused by at least one of object motion, camera motion,and a combination of object motion and camera motion.
 14. A method forextracting a still image from an input moving image, comprising stepsof: a) providing an imaging assembly including a plurality of pixels,each of the pixels comprising a photosensitive element and a pluralityof integrating storage sites, each of the pixels being constructed andarranged to direct a measurement of light impinging on thephotosensitive element to any one of the plurality of integratingstorage sites; b) forming a temporally decomposed representation of themoving image in an exposure time interval by directing, according to apredetermined temporal switching pattern, the measurement of lightimpinging on each of the photosensitive elements among the plurality ofintegrating storage sites, such that the temporally decomposed image isformed with a temporally decomposed point-spread-function having aninverse transformation that is optimally well-conditioned; and c)operating on the temporally decomposed representation of the movingimage to extract the still image.
 15. The method as set forth in claim14, further comprising selecting the temporal switching pattern to, atleast one of, (a) minimize attenuation of spatial frequencies in thetemporally decomposed representation of the moving image; (b) minimize,by the image extraction process, attenuation of spatial frequencies inthe still image; (c) provide, by the image capture process and the imageextraction process, a still image that contains a filtered set ofspatial frequencies; and (d) provide, by the image capture process andthe image extraction process a still image having reduced blur.
 16. Themethod as set forth in claim 15 wherein the step of operating on thetemporally decomposed representation includes providing a blur matrixbased upon the point-spread-function, and basing the blur matrix upon abest version of the temporal switching pattern in terms of a residualerror and a condition number.
 17. The method as set forth in claim 16wherein the step of operating on the temporally decomposedrepresentation includes determining the best version of the temporalswitching pattern by (a) using each of a plurality of candidate temporalswitching patterns to respectively generate the blur matrix; (b) solvingfor a set of blur transform coefficients with respect to each blurmatrix respectively; (c) computing respectively the residual error andthe condition number with respect to each set of blur coefficients; (d)comparing a current value of the residual error and a current value ofthe condition number to a previously existing minimum value of theresidual error and a previously existing minimal value of the conditionnumber to determine if a current one of the candidate temporal switchingpatterns represents a better solution for at least one of the residualerror and the condition number, and if the current one of the candidatetemporal switching patterns represents a better solution, saving thecurrent one of the candidate switching patterns in association with thecurrent value of the residual error and the current value of thecondition number; (e) and, when all of the candidate temporal switchingpatterns have been used, returning the best version of the temporalswitching pattern.
 18. The method as set forth in claim 14, furthercomprising capturing the moving image from a moving object operativelyconnected with an encoder that provides a signal related to movement ofthe object or scene, and basing the temporal switching pattern upon thesignal related to the movement.
 19. The method as set forth in claim 18,further comprising measuring, with an object-height-detection assembly,a height of a surface facing the imaging assembly, and based upon theheight, varying the temporal switching pattern in conjunction with thesignal related to movement of the object provided from the encoder. 20.A system for acquiring a still image from an input moving image,comprising: a) an imaging assembly including a plurality of pixels, eachof the pixels comprising a photosensitive element and a plurality ofintegrating storage sites, each of the pixels being constructed andarranged to direct a measurement of light impinging on thephotosensitive element to any one of the plurality of integratingstorage sites; b) an image capture process that forms a temporallydecomposed representation of the moving image in an exposure timeinterval by repeatedly switching the measurement of light impinging oneach of the photosensitive elements among the plurality of integratingstorage sites according to a predetermined temporal switching pattern,wherein the predetermined temporal switching pattern is the temporalswitching pattern that results in a temporally decomposedpoint-spread-function whose mathematical inverse is maximally sparse;and c) an image extraction process that operates on the temporallydecomposed representation of the moving image to extract the stillimage.